Introduction

NOTE: This is an R Markdown Notebook. When you execute code within the notebook, the results appear beneath the code.

In the previous practice you were introduced to the concept of fields/spatially continuous data.

For this practice you will need the following:

The file contains a dataframe with geocoded observations of a series of variables. The dataframe is in tidy format, that is, each observation is a row with its corresponding variables (coordinates, variables).

The Walker Lake dataset originally was used for teaching geostatistics in Isaaks and Srivastava’s An Introduction to Geostatistics.

The dataset contains a set of false coordinates X and Y, two quantitative variables V, U, and a factor variable T. The variables are generic, but you can think of them as measurements of pollutants.

In addition, the file includes two custom functions as follows:

  1. point2voronoi(sp)

This is a function to obtain Voronoi polygons based on a set of points. It takes an argument sp (a SpatialPointsDataFrame) and calculates a set of Voronoi polygons. The value (output) of the function is a SpatialPolygonsDataFrame with the polygons.

  1. kpointmeans(source_xy, z, target_xy, k, latlong)

This is a function to calculate k-point means. It takes a set of source coordinates (source_xy), that is, the coordinates of observations to be used for interpolation; a variable z to interpolate; a set of target coordinates (target_xy), the points to interpolate z; the number of nearest neighbors k; and a logical value to indicate whether the coordinates are latitude-longitude (the default is FALSE).

Learning objectives

In this practice, you will learn:

  1. About intervals of confidence for predictions.
  2. Using trend surface analysis as an interpolation tool.
  3. The difference between accuracy and precision in interpolation.

Suggested reading

O’Sullivan D and Unwin D (2010) Geographic Information Analysis, 2nd Edition, Chapters 8 and 9. John Wiley & Sons: New Jersey.

Preliminaries

As usual, it is good practice to clear the working space to make sure that you do not have extraneous items there when you begin your work. The command in R to clear the workspace is rm (for “remove”), followed by a list of items to be removed. To clear the workspace from all objects, do the following:

rm(list = ls())

Note that ls() lists all objects currently on the worspace.

Load the libraries you will use in this activity:

library(tidyverse)
-- Attaching packages --------------------------------- tidyverse 1.2.1 --
v ggplot2 2.2.1     v purrr   0.2.4
v tibble  1.4.2     v dplyr   0.7.4
v tidyr   0.8.0     v stringr 1.2.0
v readr   1.1.1     v forcats 0.2.0
-- Conflicts ------------------------------------ tidyverse_conflicts() --
x dplyr::filter() masks stats::filter()
x dplyr::lag()    masks stats::lag()
library(spdep)
Loading required package: sp
Loading required package: Matrix

Attaching package: <U+393C><U+3E31>Matrix<U+393C><U+3E32>

The following object is masked from <U+393C><U+3E31>package:tidyr<U+393C><U+3E32>:

    expand

Loading required package: spData
library(plotly)

Attaching package: <U+393C><U+3E31>plotly<U+393C><U+3E32>

The following object is masked from <U+393C><U+3E31>package:ggplot2<U+393C><U+3E32>:

    last_plot

The following object is masked from <U+393C><U+3E31>package:stats<U+393C><U+3E32>:

    filter

The following object is masked from <U+393C><U+3E31>package:graphics<U+393C><U+3E32>:

    layout
library(deldir)
deldir 0.1-14
library(spatstat)
Loading required package: spatstat.data
Loading required package: nlme

Attaching package: <U+393C><U+3E31>nlme<U+393C><U+3E32>

The following object is masked from <U+393C><U+3E31>package:dplyr<U+393C><U+3E32>:

    collapse

Loading required package: rpart

spatstat 1.55-0       (nickname: <U+393C><U+3E31>Stunned Mullet<U+393C><U+3E32>) 
For an introduction to spatstat, type <U+393C><U+3E31>beginner<U+393C><U+3E32> 

Begin by loading the data file:

load("Walker_Lake.RData")

You can verify the contents of the dataframe:

summary(Walker_Lake.t)
      ID                  X               Y               V         
 Length:470         Min.   :  8.0   Min.   :  8.0   Min.   :   0.0  
 Class :character   1st Qu.: 51.0   1st Qu.: 80.0   1st Qu.: 182.0  
 Mode  :character   Median : 89.0   Median :139.5   Median : 425.2  
                    Mean   :111.1   Mean   :141.3   Mean   : 435.4  
                    3rd Qu.:170.0   3rd Qu.:208.0   3rd Qu.: 644.4  
                    Max.   :251.0   Max.   :291.0   Max.   :1528.1  
                                                                    
       U           T      
 Min.   :   0.00   1: 45  
 1st Qu.:  83.95   2:425  
 Median : 335.00          
 Mean   : 613.27          
 3rd Qu.: 883.20          
 Max.   :5190.10          
 NA's   :195              

You can also check that the two functions have been loaded to your workspace.

Uncertainty in the predictions

In the preceding practice, you were introduced to three methods for interpolating spatially continuous data based on a sample of observations, namely Voronoi polygons, IDW, and k-point means.

These algorithms accomplish one of the objectives we typically have for spatially continuous data, since they provide point estimates. Recall that a field that was the outcome of a purely spatial process was defined as: \[ z_i = f(x_i, y_i) + \epsilon_i \]

A prediction of the field at a location where it was not measured was defined as a function of the estimated process and some random residual as follows: \[ \hat{z}_p = \hat{f}(x_p, y_p) + \hat{\epsilon}_p \] The first part of the prediction (\(\hat{f}(x_p, y_p)\)) is the point estimate of the prediction, whereas the second part (\(\hat{\epsilon}_p\)) is the random element.

The three methods seen previously do not, unfortunately, provide an estimate for the random element, so it is not possible to assess the uncertainty directly, which would depend on the distribution of the random term.

There are different ways in which some crude assessment of uncertainty could be attached to the point estimates. For example, a very simple approach could be to use the sample variance to calculate intervals of confidence. This could be done as follows.

We know that the sample variance describes the inherent variability in the distribution of values of a variable in a sample.

Consider for instance the distribution of the variable in the Walker Lake data:

ggplot(data = Walker_Lake.t, aes(V)) + geom_freqpoly()

Clearly, there are no values of the variable less than zero, and values in excess of 1,500 are rare.

The standard deviation of the sample is:

sd(Walker_Lake.t$V)
[1] 301.1554

The standard deviation is the average deviation from the mean. We could use this value to say that typical deviations from our point estimates are a function of this standard deviation (to what extent, it depends on the underlying distribution).

A problem with using this approach is that the distribution of the variable is not normal, and the distribution of \(\hat{\epsilon}_p\); the standard deviation is centered on the mean (meaning that it is a poor estimate for observations away from the mean); and in any case the standard deviation of the sample is too large for local point estimates if there is spatial pattern (since we know that the local mean will vary systematically).

There are other approaches to deal with non-normal variables, for instance Wilcox’s test, but some of the other limitations remain.

wilcox.test(Walker_Lake.t$V, conf.int = TRUE, conf.level = 0.95)

    Wilcoxon signed rank test with continuity correction

data:  Walker_Lake.t$V
V = 100580, p-value < 2.2e-16
alternative hypothesis: true location is not equal to 0
95 percent confidence interval:
 418.7 475.6
sample estimates:
(pseudo)median 
         447.2 

As an alternative, the local standard deviation could be used.

Consider the case of k-point means. The point estimate is based on the values of the k-nearest neighbors: \[ \hat{z}_p = \frac{\sum_{i=1}^n{w_{pi}z_i}}{\sum_{i=1}^n{w_{pi}}} \]

With: \[ w_{pi} = \bigg\{\begin{array}{ll} 1 & \text{if } i \text{ is one of } kth \text{ nearest neighbors of } p \text{ for a given }k \\ 0 & otherwise \\ \end{array} \]

The standard deviation could be calculated also based in the values of the k-nearest neighbors, meaning that it would be based on the local mean. Lets interpolate the field using the Walker Lake data. First create a target grid for interpolation, and extract the coordinates of observations:

target_xy = expand.grid(x = seq(0.5, 259.5, 2.2), y = seq(0.5, 299.5, 2.2))
source_xy = cbind(x = Walker_Lake.t$X, y = Walker_Lake.t$Y)

Interpolation using \(k=5\) neighbors:

kpoint.5 <- kpointmean(source_xy = source_xy, z = Walker_Lake.t$V, target_xy = target_xy, k = 5)

We can plot the interpolated field now. These are the interpolated values:

ggplot(data = kpoint.5, aes(x = x, y = y, fill = z)) +
  geom_tile() +
  scale_fill_distiller(palette = "OrRd", trans = "reverse") +
  coord_equal()

In addition, we can plot the local standard deviation:

ggplot(data = kpoint.5, aes(x = x, y = y, fill = sd)) +
  geom_tile() +
  scale_fill_distiller(palette = "OrRd", trans = "reverse") +
  coord_equal()

The local standard deviation indicates the typical deviation from the local mean. As expected, the standard deviation locally is usually lower than the standard deviation of the sample, and it tends to be larger for the tails, that is the locations where the values are rare.

The local standard deviation is a crude estimator of the uncertainty, again because we do not know the underlying distribution.

Other approaches based on bootstrapping could be implemented, but they are beyond the scope of the present discussion.

The issues assessing the level of uncertainty in the predictions with Voronoi polygons, IDW, and k-point means reflect the fact that these methods were not designed to deal explicitly with the random nature of predicting fields.

Other methods deal with this issue more naturally. We will revisit two estimation methods that were covered before, and see how they can be applied to spatial interpolation.

Trend surface analysis

Trend surface analysis is a form of multivariate regression that uses the coordinates of the observations to fit a surface to the data.

Lets see how this would work with a simulated example. We will begin by simulating a set of observations, beginning with the coordinates in the square unit region:

n <- 180
df <- data.frame(x = runif(n = n, min = 0, max = 1), 
                 y = runif(n = n, min = 0, max = 1))

Plot these locations:

ggplot(data = df, aes(x = x, y = y)) + 
  geom_point() + 
  coord_equal()

Lets now simulate a spatial process as follows:

df <- mutate(df, z = 0.5 + 0.3 * x + 0.7 * y + rnorm(n = n, mean = 0, sd = 0.1))

A 3D scatterplot can be useful to explore the data:

plot_ly(data = df, x = ~x, y = ~y, z = ~z, color = ~z) %>% 
  add_markers() %>% 
  layout(scene = list(
    aspectmode = "manual", aspectratio = list(x=1, y=1, z=1)))

We can fit a trend surface to the data as follows. In this case, the trend is linear:

trend.l <- lm(formula = z ~ x + y, data = df)
summary(trend.l)

Call:
lm(formula = z ~ x + y, data = df)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.194786 -0.064316  0.005245  0.058750  0.230605 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  0.50496    0.01975   25.57   <2e-16 ***
x            0.32199    0.02496   12.90   <2e-16 ***
y            0.69977    0.02461   28.44   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.09631 on 177 degrees of freedom
Multiple R-squared:  0.8434,    Adjusted R-squared:  0.8417 
F-statistic: 476.7 on 2 and 177 DF,  p-value: < 2.2e-16

Given a trend surface model, we can estimate the value of the variable \(z\) at locations where it was not measured. Typically this is done by interpolating on a fine grid that can be used for vizualization or further analysis, as shown next.

We will begin by creating a grid for interpolation. We will call the coordinates x.p and y.p. These we generate by creating a sequence of values in the domain of the data, for instance in the [0,1] interval:

x.p <- seq(from = 0.0, to = 1.0, by = 0.05)
y.p <- seq(from = 0.0, to = 1.0, by = 0.05)

For prediction, we want all combinations of x.p and y.p, so we expand these two vectors into a grid, by means of the function expand.grid:

df.p <- expand.grid(x = x.p, y = y.p)

Notice that while x.p and y.p are of vectors of size 21, the dataframe df.p contains {r}21 * 21 observations.

Once that we have the coordinates for interpolation, the predict function can be used in conjunction with the results of the estimation. When invoking the function, we indicate that we wish to obtain as well the standard errors of the fitted values (se.fit = TRUE), as well as the interval of the predictions at a 95% level of confidence:

preds <- predict(trend.l, newdata = df.p, se.fit = TRUE, interval = "prediction", level = 0.95)

The interval of confidence of the predictions at the 95% level of confidence, in the form of the lower (lwr) and upper (upr) bounds:

summary(preds$fit)
      fit              lwr              upr       
 Min.   :0.5050   Min.   :0.3109   Min.   :0.699  
 1st Qu.:0.8325   1st Qu.:0.6404   1st Qu.:1.025  
 Median :1.0158   Median :0.8253   Median :1.206  
 Mean   :1.0158   Mean   :0.8241   Mean   :1.208  
 3rd Qu.:1.1992   3rd Qu.:1.0072   3rd Qu.:1.391  
 Max.   :1.5267   Max.   :1.3331   Max.   :1.720  

These values indicate that the predictions of \(z_p\) are, with 95% of confidence, in the following interval: \[ CI_{z_p} = [z_{p_{lwr}}, z_{p_{upr}}]. \]

A convenient way to visualize the results of the analysis above, that is, to inspect the trend surface and the interval of confidence of the predictions, is by means of a 3D plot as follows.

First create matrices with the point estimates of the trend surface (z.p), and the lower and upper bounds (z.p_l, z.p_u):

z.p <- matrix(data = preds$fit[,1], nrow = 21, ncol = 21, byrow = TRUE)
z.p_l <- matrix(data = preds$fit[,2], nrow = 21, ncol = 21, byrow = TRUE)
z.p_u <- matrix(data = preds$fit[,3], nrow = 21, ncol = 21, byrow = TRUE)

The plot is created using the coordinates used for interpolation (x.p and y.p) and the matrices with the point estimates z.p and the upper and lower bounds. The type of plot in the package plotly is a surface:

trend.plot <- plot_ly(x = ~x.p, y = ~y.p, z = ~z.p, 
        type = "surface", colors = "YlOrRd") %>% 
  add_surface(x = ~x.p, y = ~y.p, z = ~z.p_l, 
              opacity = 0.5, showscale = FALSE) %>%
  add_surface(x = ~x.p, y = ~y.p, z = ~z.p_u, 
              opacity = 0.5, showscale = FALSE) %>% 
  layout(scene = list(
    aspectmode = "manual", aspectratio = list(x = 1, y = 1, z = 1)))
trend.plot #%>% add_markers(data = df, x = ~x, y = ~y, z = ~z, color = ~residual)

In this way, we have not only an estimate of the underlying field, but also a measure of uncertainty for our predictions, since our estimated values are bound, with 95% confidence, between the lower and upper surfaces.

It is important to note that, although the confidence interval provides a measure of uncertainty, it does not provide an estimate of the prediction error \(\hat{\epsilon}_p\). This quantity cannot be calculated directly, because we do not know the true value of the field at location \(p\). We will revisit this point later.

For the time being, lets apply trend surface analysis to the Walker Lake data.

We will first calculate the polynomial terms of the coordinates, for instance to the 3rd degree (this can be done to any arbitrary degree, however keeping in mind the caveats discussed previously with respect to trend surface analysis):

Walker_Lake.t <- mutate(Walker_Lake.t,
                        X3 = X^3, X2Y = X^2 * Y, X2 = X^2, 
                        XY = X * Y,
                        Y2 = Y^2, XY2 = X * Y^2, Y3 = Y^3)

We can proceed to estimate the following models.

Linear trend surface model:

WL.trend1 <- lm(formula = V ~ X + Y, data = Walker_Lake.t)
summary(WL.trend1)

Call:
lm(formula = V ~ X + Y, data = Walker_Lake.t)

Residuals:
    Min      1Q  Median      3Q     Max 
-576.05 -241.79   -4.77  201.48 1055.98 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 589.5289    34.5547  17.061  < 2e-16 ***
X            -1.0082     0.1903  -5.297 1.82e-07 ***
Y            -0.2980     0.1727  -1.726   0.0851 .  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 292.1 on 467 degrees of freedom
Multiple R-squared:  0.06338,   Adjusted R-squared:  0.05937 
F-statistic:  15.8 on 2 and 467 DF,  p-value: 2.29e-07

Quadratic trend surface model:

WL.trend2 <- lm(formula = V ~ X2 + X + XY + Y + Y2, data = Walker_Lake.t)
summary(WL.trend2)

Call:
lm(formula = V ~ X2 + X + XY + Y + Y2, data = Walker_Lake.t)

Residuals:
    Min      1Q  Median      3Q     Max 
-579.37 -220.93    3.96  200.66  997.30 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) 320.484250  70.904832   4.520 7.86e-06 ***
X2            0.001145   0.003191   0.359  0.71994    
X            -0.325993   0.910872  -0.358  0.72059    
XY           -0.006281   0.002244  -2.799  0.00534 ** 
Y             3.737955   0.722097   5.177 3.37e-07 ***
Y2           -0.011409   0.002188  -5.215 2.77e-07 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 283.1 on 464 degrees of freedom
Multiple R-squared:  0.1258,    Adjusted R-squared:  0.1164 
F-statistic: 13.35 on 5 and 464 DF,  p-value: 3.571e-12

Cubic trend surface model:

WL.trend3 <- lm(formula = V ~ X3 + X2Y + X2 + X + XY + Y + Y2 + XY2 + Y3, 
                data = Walker_Lake.t)
summary(WL.trend3)

Call:
lm(formula = V ~ X3 + X2Y + X2 + X + XY + Y + Y2 + XY2 + Y3, 
    data = Walker_Lake.t)

Residuals:
    Min      1Q  Median      3Q     Max 
-564.19 -197.41    7.91  194.25  929.72 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -8.620e+00  1.227e+02  -0.070 0.944035    
X3           1.533e-04  4.806e-05   3.190 0.001522 ** 
X2Y          6.139e-05  3.909e-05   1.570 0.117000    
X2          -6.651e-02  1.838e-02  -3.618 0.000330 ***
X            9.172e+00  2.386e+00   3.844 0.000138 ***
XY          -4.420e-02  1.430e-02  -3.092 0.002110 ** 
Y            4.794e+00  2.040e+00   2.350 0.019220 *  
Y2          -1.806e-03  1.327e-02  -0.136 0.891822    
XY2          7.679e-05  2.956e-05   2.598 0.009669 ** 
Y3          -4.170e-05  2.819e-05  -1.479 0.139759    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 276.7 on 460 degrees of freedom
Multiple R-squared:  0.1719,    Adjusted R-squared:  0.1557 
F-statistic: 10.61 on 9 and 460 DF,  p-value: 5.381e-15

Inspection of the results of the three models above suggests that the cubic trend surface provides the best fit, with the highest adjusted coefficient of determination, even if the value is relatively low at approximately 0.16. Also, the cubic trend yields the smallest standard error, which implies that the intervals of confidence are tighter, and hence the degree of uncertainty is smaller.

Lets compare two of these models to see how well they fit the data.

First, we use create an interpolation grid. Summarize the information to ascertain the domain of the data:

summary(Walker_Lake.t[,2:3])
       X               Y        
 Min.   :  8.0   Min.   :  8.0  
 1st Qu.: 51.0   1st Qu.: 80.0  
 Median : 89.0   Median :139.5  
 Mean   :111.1   Mean   :141.3  
 3rd Qu.:170.0   3rd Qu.:208.0  
 Max.   :251.0   Max.   :291.0  

We can see that the spatial domain is in the range of [8,251] in X, and [8,291] in Y. Based on this information, we generate the following sequence that then is expanded into a grid for prediction:

X.p <- seq(from = 0.0, to = 255.0, by = 2.5)
Y.p <- seq(from = 0.0, to = 295.0, by = 2.5)
df.p <- expand.grid(X = X.p, Y = Y.p)

To this dataframe we add the polynomial terms:

df.p <- mutate(df.p, X3 = X^3, X2Y = X^2 * Y, X2 = X^2, 
               XY = X * Y, 
               Y2 = Y^2, XY2 = X * Y^2, Y3 = Y^3)

The interpolated quadratic surface is then obtained as:

WL.preds2 <- predict(WL.trend2, newdata = df.p, se.fit = TRUE, interval = "prediction", level = 0.95)

Whereas the interpolated cubic surface is obtained as:

WL.preds3 <- predict(WL.trend3, newdata = df.p, se.fit = TRUE, interval = "prediction", level = 0.95)

The predictions are transformed into matrices for ploting.

Quadratic trend surface and lower and upper bounds of the predictions:

z.p2 <- matrix(data = WL.preds2$fit[,1], nrow = 119, ncol = 103, byrow = TRUE)
z.p2_l <- matrix(data = WL.preds2$fit[,2], nrow = 119, ncol = 103, byrow = TRUE)
z.p2_u <- matrix(data = WL.preds2$fit[,3], nrow = 119, ncol = 103, byrow = TRUE)

Cubic trend surface and lower and upper bounds of the predictions:

z.p3 <- matrix(data = WL.preds3$fit[,1], nrow = 119, ncol = 103, byrow = TRUE)
z.p3_l <- matrix(data = WL.preds3$fit[,2], nrow = 119, ncol = 103, byrow = TRUE)
z.p3_u <- matrix(data = WL.preds3$fit[,3], nrow = 119, ncol = 103, byrow = TRUE)

This is the quadratic trend surface with its confidence interval of predictions:

WL.plot2 <- plot_ly(x = ~X.p, y = ~Y.p, z = ~z.p2, 
        type = "surface", colors = "YlOrRd") %>% 
  add_surface(x = ~X.p, y = ~Y.p, z = ~z.p2_l, 
              opacity = 0.5, showscale = FALSE) %>%
  add_surface(x = ~X.p, y = ~Y.p, z = ~z.p2_u, 
              opacity = 0.5, showscale = FALSE) %>% 
  layout(scene = list(
    aspectmode = "manual", aspectratio = list(x = 1, y = 1, z = 1)))
WL.plot2

And, this is the cubic trend surface with its confidence interval of predictions:

WL.plot3 <- plot_ly(x = ~X.p, y = ~Y.p, z = ~z.p3, 
        type = "surface", colors = "YlOrRd") %>% 
  add_surface(x = ~X.p, y = ~Y.p, z = ~z.p3_l, 
              opacity = 0.5, showscale = FALSE) %>%
  add_surface(x = ~X.p, y = ~Y.p, z = ~z.p3_u, 
              opacity = 0.5, showscale = FALSE) %>% 
  layout(scene = list(
    aspectmode = "manual", aspectratio = list(x = 1, y = 1, z = 1)))
WL.plot3

Alas, these models are not very reliable estimates of the underlying field. As can be seen from the plots, the confidence intervals are extremely wide, and in both cases include negative numbers in the lower bound. The uncertainty associated with these predictions is quite substantial.

Another question, however, is whether the point estimates are accurate. Lets add the observations to the plot:

WL.plot3 %>%
  add_markers(data = Walker_Lake.t, x = ~X, y = ~Y, z = ~V, 
              color = ~V, opacity = 0.7, showlegend = FALSE)

Alas, the trend surface does a mediocre job with the point estimates as well.

A possible reason for this is that the model failed to capture the spatial variability. To explore this, we will plot the residuals of the model, after labeling them as “positive” or “negative”:

Walker_Lake.t$residual3 <- ifelse(WL.trend3$residuals > 0, "Positive", "Negative")
ggplot(data = Walker_Lake.t, 
       aes(x = X, y = Y, color = residual3)) +
  geom_point() +
  coord_equal()

Visual inspection of the distribution of the residuals strongly suggests that they are not random. We can check this by means of Moran’s I coefficient, if we create a list of spatial weights as follows:

WL.listw <- nb2listw(knn2nb(knearneigh(as.matrix(Walker_Lake.t[,2:3]), k = 5)))

The results of the autocorrelation analysis of the residuals are:

moran.test(x = WL.trend3$residuals, listw = WL.listw)

    Moran I test under randomisation

data:  WL.trend3$residuals  
weights: WL.listw  

Moran I statistic standard deviate = 17.199, p-value < 2.2e-16
alternative hypothesis: greater
sample estimates:
Moran I statistic       Expectation          Variance 
     0.4633803457     -0.0021321962      0.0007325452 

Given the low p-value, we fail to reject the null hypothesis, and conclude, with a high level of confidence, that the residuals are not independent. This has important implications for spatial interpolation, as we will discuss in the following practice.

Accuracy and precision

Before concluding this practice, it is worthwhile to make the following distinction between accuracy and precision of the estimates.

Accuracy refers to how close the predicted values \(\hat{z}_p\) are to the true values of the field. Precision refers to how much uncertainty is associated with such predictions. Narrow intervals of confidence imply greater precision, whereas the opposite is true when the intervals of confidence are wide.

An example of these two properties is as shown in Figure 1.

Figure 1. Accuracy and precision

Figure 1. Accuracy and precision

Panel a) in the figure represents a set of accurate points, since they are on average close to the mark. However, they are imprecise, given their variability. This is akin to a good point estimate that has wide confidence intervals.

Panel b) is a set of inaccurate and imprecise points.

Panel c) is a set of precise but inaccurate points.

Finally, Panel d) is a set of accurate and precise points.

Accuracy and precision are important criteria when assessing the quality of a predictive model.

This concludes Practice 15.

LS0tDQp0aXRsZTogIjE2IFNwYXRpYWxseSBDb250aW51b3VzIERhdGEgSUkiDQpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sNCi0tLQ0KDQojSW50cm9kdWN0aW9uDQoNCk5PVEU6IFRoaXMgaXMgYW4gW1IgTWFya2Rvd25dKGh0dHA6Ly9ybWFya2Rvd24ucnN0dWRpby5jb20pIE5vdGVib29rLiBXaGVuIHlvdSBleGVjdXRlIGNvZGUgd2l0aGluIHRoZSBub3RlYm9vaywgdGhlIHJlc3VsdHMgYXBwZWFyIGJlbmVhdGggdGhlIGNvZGUuIA0KDQpJbiB0aGUgcHJldmlvdXMgcHJhY3RpY2UgeW91IHdlcmUgaW50cm9kdWNlZCB0byB0aGUgY29uY2VwdCBvZiBmaWVsZHMvc3BhdGlhbGx5IGNvbnRpbnVvdXMgZGF0YS4NCg0KRm9yIHRoaXMgcHJhY3RpY2UgeW91IHdpbGwgbmVlZCB0aGUgZm9sbG93aW5nOg0KDQoqIFRoaXMgUiBtYXJrZG93biBub3RlYm9vay4NCiogQSBmaWxlIGNhbGxlZCAiV2Fsa2VyX0xha2UuUkRhdGEiDQoNClRoZSBmaWxlIGNvbnRhaW5zIGEgZGF0YWZyYW1lIHdpdGggZ2VvY29kZWQgb2JzZXJ2YXRpb25zIG9mIGEgc2VyaWVzIG9mIHZhcmlhYmxlcy4gVGhlIGRhdGFmcmFtZSBpcyBpbiB0aWR5IGZvcm1hdCwgdGhhdCBpcywgZWFjaCBvYnNlcnZhdGlvbiBpcyBhIHJvdyB3aXRoIGl0cyBjb3JyZXNwb25kaW5nIHZhcmlhYmxlcyAoY29vcmRpbmF0ZXMsIHZhcmlhYmxlcykuDQoNClRoZSBXYWxrZXIgTGFrZSBkYXRhc2V0IG9yaWdpbmFsbHkgd2FzIHVzZWQgZm9yIHRlYWNoaW5nIGdlb3N0YXRpc3RpY3MgaW4gSXNhYWtzIGFuZCBTcml2YXN0YXZhJ3MgW0FuIEludHJvZHVjdGlvbiB0byBHZW9zdGF0aXN0aWNzXShodHRwczovL2Jvb2tzLmdvb2dsZS5jYS9ib29rcz9pZD12QzJkY1hGTEkzWUMmZHE9aW50cm9kdWN0aW9uK3RvK2FwcGxpZWQrZ2Vvc3RhdGlzdGljcytpc2Fha3MrYW5kK3NyaXZhc3RhdmEmaGw9ZW4mc2E9WCZ2ZWQ9MGFoVUtFd2lLZzZfaXlyWFpBaFVqcDFrS0hkX2pBVmNRNkFFSUtUQUEpLg0KDQpUaGUgZGF0YXNldCBjb250YWlucyBhIHNldCBvZiBmYWxzZSBjb29yZGluYXRlcyBgWGAgYW5kIGBZYCwgdHdvIHF1YW50aXRhdGl2ZSB2YXJpYWJsZXMgYFZgLCBgVWAsIGFuZCBhIGZhY3RvciB2YXJpYWJsZSBgVGAuIFRoZSB2YXJpYWJsZXMgYXJlIGdlbmVyaWMsIGJ1dCB5b3UgY2FuIHRoaW5rIG9mIHRoZW0gYXMgbWVhc3VyZW1lbnRzIG9mIHBvbGx1dGFudHMuDQoNCkluIGFkZGl0aW9uLCB0aGUgZmlsZSBpbmNsdWRlcyB0d28gY3VzdG9tIGZ1bmN0aW9ucyBhcyBmb2xsb3dzOg0KDQoxLiBwb2ludDJ2b3Jvbm9pKHNwKQ0KDQpUaGlzIGlzIGEgZnVuY3Rpb24gdG8gb2J0YWluIFZvcm9ub2kgcG9seWdvbnMgYmFzZWQgb24gYSBzZXQgb2YgcG9pbnRzLiBJdCB0YWtlcyBhbiBhcmd1bWVudCBgc3BgIChhIGBTcGF0aWFsUG9pbnRzRGF0YUZyYW1lYCkgYW5kIGNhbGN1bGF0ZXMgYSBzZXQgb2YgVm9yb25vaSBwb2x5Z29ucy4gVGhlIHZhbHVlIChvdXRwdXQpIG9mIHRoZSBmdW5jdGlvbiBpcyBhIGBTcGF0aWFsUG9seWdvbnNEYXRhRnJhbWVgIHdpdGggdGhlIHBvbHlnb25zLg0KDQoyLiBrcG9pbnRtZWFucyhzb3VyY2VfeHksIHosIHRhcmdldF94eSwgaywgbGF0bG9uZykNCg0KVGhpcyBpcyBhIGZ1bmN0aW9uIHRvIGNhbGN1bGF0ZSBrLXBvaW50IG1lYW5zLiBJdCB0YWtlcyBhIHNldCBvZiBzb3VyY2UgY29vcmRpbmF0ZXMgKGBzb3VyY2VfeHlgKSwgdGhhdCBpcywgdGhlIGNvb3JkaW5hdGVzIG9mIG9ic2VydmF0aW9ucyB0byBiZSB1c2VkIGZvciBpbnRlcnBvbGF0aW9uOyBhIHZhcmlhYmxlIGB6YCB0byBpbnRlcnBvbGF0ZTsgYSBzZXQgb2YgdGFyZ2V0IGNvb3JkaW5hdGVzIChgdGFyZ2V0X3h5YCksIHRoZSBwb2ludHMgdG8gaW50ZXJwb2xhdGUgYHpgOyB0aGUgbnVtYmVyIG9mIG5lYXJlc3QgbmVpZ2hib3JzIGBrYDsgYW5kIGEgbG9naWNhbCB2YWx1ZSB0byBpbmRpY2F0ZSB3aGV0aGVyIHRoZSBjb29yZGluYXRlcyBhcmUgbGF0aXR1ZGUtbG9uZ2l0dWRlICh0aGUgZGVmYXVsdCBpcyBgRkFMU0VgKS4NCg0KI0xlYXJuaW5nIG9iamVjdGl2ZXMNCg0KSW4gdGhpcyBwcmFjdGljZSwgeW91IHdpbGwgbGVhcm46DQoNCjEuIEFib3V0IGludGVydmFscyBvZiBjb25maWRlbmNlIGZvciBwcmVkaWN0aW9ucy4NCjIuIFVzaW5nIHRyZW5kIHN1cmZhY2UgYW5hbHlzaXMgYXMgYW4gaW50ZXJwb2xhdGlvbiB0b29sLg0KMy4gVGhlIGRpZmZlcmVuY2UgYmV0d2VlbiBhY2N1cmFjeSBhbmQgcHJlY2lzaW9uIGluIGludGVycG9sYXRpb24uDQoNCiNTdWdnZXN0ZWQgcmVhZGluZw0KDQpPJ1N1bGxpdmFuIEQgYW5kIFVud2luIEQgKDIwMTApIEdlb2dyYXBoaWMgSW5mb3JtYXRpb24gQW5hbHlzaXMsIDJuZCBFZGl0aW9uLCBDaGFwdGVycyA4IGFuZCA5LiBKb2huIFdpbGV5ICYgU29uczogTmV3IEplcnNleS4NCg0KI1ByZWxpbWluYXJpZXMNCg0KQXMgdXN1YWwsIGl0IGlzIGdvb2QgcHJhY3RpY2UgdG8gY2xlYXIgdGhlIHdvcmtpbmcgc3BhY2UgdG8gbWFrZSBzdXJlIHRoYXQgeW91IGRvIG5vdCBoYXZlIGV4dHJhbmVvdXMgaXRlbXMgdGhlcmUgd2hlbiB5b3UgYmVnaW4geW91ciB3b3JrLiBUaGUgY29tbWFuZCBpbiBSIHRvIGNsZWFyIHRoZSB3b3Jrc3BhY2UgaXMgYHJtYCAoZm9yICJyZW1vdmUiKSwgZm9sbG93ZWQgYnkgYSBsaXN0IG9mIGl0ZW1zIHRvIGJlIHJlbW92ZWQuIFRvIGNsZWFyIHRoZSB3b3Jrc3BhY2UgZnJvbSBfYWxsXyBvYmplY3RzLCBkbyB0aGUgZm9sbG93aW5nOg0KYGBge3J9DQpybShsaXN0ID0gbHMoKSkNCmBgYA0KDQpOb3RlIHRoYXQgYGxzKClgIGxpc3RzIGFsbCBvYmplY3RzIGN1cnJlbnRseSBvbiB0aGUgd29yc3BhY2UuDQoNCkxvYWQgdGhlIGxpYnJhcmllcyB5b3Ugd2lsbCB1c2UgaW4gdGhpcyBhY3Rpdml0eToNCmBgYHtyfQ0KbGlicmFyeSh0aWR5dmVyc2UpDQpsaWJyYXJ5KHNwZGVwKQ0KbGlicmFyeShwbG90bHkpDQpsaWJyYXJ5KGRlbGRpcikNCmxpYnJhcnkoc3BhdHN0YXQpDQpgYGANCg0KQmVnaW4gYnkgbG9hZGluZyB0aGUgZGF0YSBmaWxlOg0KYGBge3J9DQpsb2FkKCJXYWxrZXJfTGFrZS5SRGF0YSIpDQpgYGANCg0KWW91IGNhbiB2ZXJpZnkgdGhlIGNvbnRlbnRzIG9mIHRoZSBkYXRhZnJhbWU6DQpgYGB7cn0NCnN1bW1hcnkoV2Fsa2VyX0xha2UudCkNCmBgYA0KDQpZb3UgY2FuIGFsc28gY2hlY2sgdGhhdCB0aGUgdHdvIGZ1bmN0aW9ucyBoYXZlIGJlZW4gbG9hZGVkIHRvIHlvdXIgd29ya3NwYWNlLg0KDQojIFVuY2VydGFpbnR5IGluIHRoZSBwcmVkaWN0aW9ucw0KDQpJbiB0aGUgcHJlY2VkaW5nIHByYWN0aWNlLCB5b3Ugd2VyZSBpbnRyb2R1Y2VkIHRvIHRocmVlIG1ldGhvZHMgZm9yIGludGVycG9sYXRpbmcgc3BhdGlhbGx5IGNvbnRpbnVvdXMgZGF0YSBiYXNlZCBvbiBhIHNhbXBsZSBvZiBvYnNlcnZhdGlvbnMsIG5hbWVseSBWb3Jvbm9pIHBvbHlnb25zLCBJRFcsIGFuZCBrLXBvaW50IG1lYW5zLg0KDQpUaGVzZSBhbGdvcml0aG1zIGFjY29tcGxpc2ggb25lIG9mIHRoZSBvYmplY3RpdmVzIHdlIHR5cGljYWxseSBoYXZlIGZvciBzcGF0aWFsbHkgY29udGludW91cyBkYXRhLCBzaW5jZSB0aGV5IHByb3ZpZGUgcG9pbnQgZXN0aW1hdGVzLiBSZWNhbGwgdGhhdCBhIGZpZWxkIHRoYXQgd2FzIHRoZSBvdXRjb21lIG9mIGEgcHVyZWx5IHNwYXRpYWwgcHJvY2VzcyB3YXMgZGVmaW5lZCBhczoNCiQkDQp6X2kgPSBmKHhfaSwgeV9pKSArIFxlcHNpbG9uX2kNCiQkDQoNCkEgcHJlZGljdGlvbiBvZiB0aGUgZmllbGQgYXQgYSBsb2NhdGlvbiB3aGVyZSBpdCB3YXMgbm90IG1lYXN1cmVkIHdhcyBkZWZpbmVkIGFzIGEgZnVuY3Rpb24gb2YgdGhlIGVzdGltYXRlZCBwcm9jZXNzIGFuZCBzb21lIHJhbmRvbSByZXNpZHVhbCBhcyBmb2xsb3dzOg0KJCQNClxoYXR7en1fcCA9IFxoYXR7Zn0oeF9wLCB5X3ApICsgXGhhdHtcZXBzaWxvbn1fcA0KJCQNClRoZSBmaXJzdCBwYXJ0IG9mIHRoZSBwcmVkaWN0aW9uICgkXGhhdHtmfSh4X3AsIHlfcCkkKSBpcyB0aGUgcG9pbnQgZXN0aW1hdGUgb2YgdGhlIHByZWRpY3Rpb24sIHdoZXJlYXMgdGhlIHNlY29uZCBwYXJ0ICgkXGhhdHtcZXBzaWxvbn1fcCQpIGlzIHRoZSByYW5kb20gZWxlbWVudC4NCg0KVGhlIHRocmVlIG1ldGhvZHMgc2VlbiBwcmV2aW91c2x5IGRvIG5vdCwgdW5mb3J0dW5hdGVseSwgcHJvdmlkZSBhbiBlc3RpbWF0ZSBmb3IgdGhlIHJhbmRvbSBlbGVtZW50LCBzbyBpdCBpcyBub3QgcG9zc2libGUgdG8gYXNzZXNzIHRoZSB1bmNlcnRhaW50eSBkaXJlY3RseSwgd2hpY2ggd291bGQgZGVwZW5kIG9uIHRoZSBkaXN0cmlidXRpb24gb2YgdGhlIHJhbmRvbSB0ZXJtLg0KDQpUaGVyZSBhcmUgZGlmZmVyZW50IHdheXMgaW4gd2hpY2ggc29tZSBjcnVkZSBhc3Nlc3NtZW50IG9mIHVuY2VydGFpbnR5IGNvdWxkIGJlIGF0dGFjaGVkIHRvIHRoZSBwb2ludCBlc3RpbWF0ZXMuIEZvciBleGFtcGxlLCBhIHZlcnkgc2ltcGxlIGFwcHJvYWNoIGNvdWxkIGJlIHRvIHVzZSB0aGUgc2FtcGxlIHZhcmlhbmNlIHRvIGNhbGN1bGF0ZSBpbnRlcnZhbHMgb2YgY29uZmlkZW5jZS4gVGhpcyBjb3VsZCBiZSBkb25lIGFzIGZvbGxvd3MuDQoNCldlIGtub3cgdGhhdCB0aGUgc2FtcGxlIHZhcmlhbmNlIGRlc2NyaWJlcyB0aGUgaW5oZXJlbnQgdmFyaWFiaWxpdHkgaW4gdGhlIGRpc3RyaWJ1dGlvbiBvZiB2YWx1ZXMgb2YgYSB2YXJpYWJsZSBpbiBhIHNhbXBsZS4NCg0KQ29uc2lkZXIgZm9yIGluc3RhbmNlIHRoZSBkaXN0cmlidXRpb24gb2YgdGhlIHZhcmlhYmxlIGluIHRoZSBXYWxrZXIgTGFrZSBkYXRhOg0KYGBge3J9DQpnZ3Bsb3QoZGF0YSA9IFdhbGtlcl9MYWtlLnQsIGFlcyhWKSkgKyBnZW9tX2ZyZXFwb2x5KCkNCmBgYA0KDQpDbGVhcmx5LCB0aGVyZSBhcmUgbm8gdmFsdWVzIG9mIHRoZSB2YXJpYWJsZSBsZXNzIHRoYW4gemVybywgYW5kIHZhbHVlcyBpbiBleGNlc3Mgb2YgMSw1MDAgYXJlIHJhcmUuDQoNClRoZSBzdGFuZGFyZCBkZXZpYXRpb24gb2YgdGhlIHNhbXBsZSBpczoNCmBgYHtyfQ0Kc2QoV2Fsa2VyX0xha2UudCRWKQ0KYGBgDQoNClRoZSBzdGFuZGFyZCBkZXZpYXRpb24gaXMgdGhlIGF2ZXJhZ2UgZGV2aWF0aW9uIGZyb20gdGhlIG1lYW4uIFdlIGNvdWxkIHVzZSB0aGlzIHZhbHVlIHRvIHNheSB0aGF0IHR5cGljYWwgZGV2aWF0aW9ucyBmcm9tIG91ciBwb2ludCBlc3RpbWF0ZXMgYXJlIGEgZnVuY3Rpb24gb2YgdGhpcyBzdGFuZGFyZCBkZXZpYXRpb24gKHRvIHdoYXQgZXh0ZW50LCBpdCBkZXBlbmRzIG9uIHRoZSB1bmRlcmx5aW5nIGRpc3RyaWJ1dGlvbikuDQoNCkEgcHJvYmxlbSB3aXRoIHVzaW5nIHRoaXMgYXBwcm9hY2ggaXMgdGhhdCB0aGUgZGlzdHJpYnV0aW9uIG9mIHRoZSB2YXJpYWJsZSBpcyBub3Qgbm9ybWFsLCBhbmQgdGhlIGRpc3RyaWJ1dGlvbiBvZiAkXGhhdHtcZXBzaWxvbn1fcCQ7IHRoZSBzdGFuZGFyZCBkZXZpYXRpb24gaXMgY2VudGVyZWQgb24gdGhlIG1lYW4gKG1lYW5pbmcgdGhhdCBpdCBpcyBhIHBvb3IgZXN0aW1hdGUgZm9yIG9ic2VydmF0aW9ucyBhd2F5IGZyb20gdGhlIG1lYW4pOyBhbmQgaW4gYW55IGNhc2UgdGhlIHN0YW5kYXJkIGRldmlhdGlvbiBvZiB0aGUgc2FtcGxlIGlzIHRvbyBsYXJnZSBmb3IgbG9jYWwgcG9pbnQgZXN0aW1hdGVzIGlmIHRoZXJlIGlzIHNwYXRpYWwgcGF0dGVybiAoc2luY2Ugd2Uga25vdyB0aGF0IHRoZSBsb2NhbCBtZWFuIHdpbGwgdmFyeSBzeXN0ZW1hdGljYWxseSkuDQoNClRoZXJlIGFyZSBvdGhlciBhcHByb2FjaGVzIHRvIGRlYWwgd2l0aCBub24tbm9ybWFsIHZhcmlhYmxlcywgZm9yIGluc3RhbmNlIFdpbGNveCdzIHRlc3QsIGJ1dCBzb21lIG9mIHRoZSBvdGhlciBsaW1pdGF0aW9ucyByZW1haW4uDQpgYGB7cn0NCndpbGNveC50ZXN0KFdhbGtlcl9MYWtlLnQkViwgY29uZi5pbnQgPSBUUlVFLCBjb25mLmxldmVsID0gMC45NSkNCmBgYA0KDQpBcyBhbiBhbHRlcm5hdGl2ZSwgdGhlIF9sb2NhbF8gc3RhbmRhcmQgZGV2aWF0aW9uIGNvdWxkIGJlIHVzZWQuDQoNCkNvbnNpZGVyIHRoZSBjYXNlIG9mIGstcG9pbnQgbWVhbnMuIFRoZSBwb2ludCBlc3RpbWF0ZSBpcyBiYXNlZCBvbiB0aGUgdmFsdWVzIG9mIHRoZSBrLW5lYXJlc3QgbmVpZ2hib3JzOg0KJCQNClxoYXR7en1fcCA9IFxmcmFje1xzdW1fe2k9MX1ebnt3X3twaX16X2l9fXtcc3VtX3tpPTF9Xm57d197cGl9fX0NCiQkDQoNCldpdGg6DQokJA0Kd197cGl9ID0gXGJpZ2dce1xiZWdpbnthcnJheX17bGx9DQoxICYgXHRleHR7aWYgfSBpIFx0ZXh0eyBpcyBvbmUgb2YgfSBrdGggXHRleHR7IG5lYXJlc3QgbmVpZ2hib3JzIG9mIH0gcCBcdGV4dHsgZm9yIGEgZ2l2ZW4gfWsgXFwNCjAgJiBvdGhlcndpc2UgXFwNClxlbmR7YXJyYXl9DQokJA0KDQpUaGUgc3RhbmRhcmQgZGV2aWF0aW9uIGNvdWxkIGJlIGNhbGN1bGF0ZWQgYWxzbyBiYXNlZCBpbiB0aGUgdmFsdWVzIG9mIHRoZSBrLW5lYXJlc3QgbmVpZ2hib3JzLCBtZWFuaW5nIHRoYXQgaXQgd291bGQgYmUgYmFzZWQgb24gdGhlIGxvY2FsIG1lYW4uIExldHMgaW50ZXJwb2xhdGUgdGhlIGZpZWxkIHVzaW5nIHRoZSBXYWxrZXIgTGFrZSBkYXRhLiBGaXJzdCBjcmVhdGUgYSB0YXJnZXQgZ3JpZCBmb3IgaW50ZXJwb2xhdGlvbiwgYW5kIGV4dHJhY3QgdGhlIGNvb3JkaW5hdGVzIG9mIG9ic2VydmF0aW9uczogDQpgYGB7cn0NCnRhcmdldF94eSA9IGV4cGFuZC5ncmlkKHggPSBzZXEoMC41LCAyNTkuNSwgMi4yKSwgeSA9IHNlcSgwLjUsIDI5OS41LCAyLjIpKQ0Kc291cmNlX3h5ID0gY2JpbmQoeCA9IFdhbGtlcl9MYWtlLnQkWCwgeSA9IFdhbGtlcl9MYWtlLnQkWSkNCmBgYA0KDQpJbnRlcnBvbGF0aW9uIHVzaW5nICRrPTUkIG5laWdoYm9yczoNCmBgYHtyfQ0Ka3BvaW50LjUgPC0ga3BvaW50bWVhbihzb3VyY2VfeHkgPSBzb3VyY2VfeHksIHogPSBXYWxrZXJfTGFrZS50JFYsIHRhcmdldF94eSA9IHRhcmdldF94eSwgayA9IDUpDQpgYGANCg0KV2UgY2FuIHBsb3QgdGhlIGludGVycG9sYXRlZCBmaWVsZCBub3cuIFRoZXNlIGFyZSB0aGUgaW50ZXJwb2xhdGVkIHZhbHVlczoNCmBgYHtyfQ0KZ2dwbG90KGRhdGEgPSBrcG9pbnQuNSwgYWVzKHggPSB4LCB5ID0geSwgZmlsbCA9IHopKSArDQogIGdlb21fdGlsZSgpICsNCiAgc2NhbGVfZmlsbF9kaXN0aWxsZXIocGFsZXR0ZSA9ICJPclJkIiwgdHJhbnMgPSAicmV2ZXJzZSIpICsNCiAgY29vcmRfZXF1YWwoKQ0KYGBgDQoNCkluIGFkZGl0aW9uLCB3ZSBjYW4gcGxvdCB0aGUgX2xvY2FsXyBzdGFuZGFyZCBkZXZpYXRpb246DQpgYGB7cn0NCmdncGxvdChkYXRhID0ga3BvaW50LjUsIGFlcyh4ID0geCwgeSA9IHksIGZpbGwgPSBzZCkpICsNCiAgZ2VvbV90aWxlKCkgKw0KICBzY2FsZV9maWxsX2Rpc3RpbGxlcihwYWxldHRlID0gIk9yUmQiLCB0cmFucyA9ICJyZXZlcnNlIikgKw0KICBjb29yZF9lcXVhbCgpDQpgYGANCg0KVGhlIGxvY2FsIHN0YW5kYXJkIGRldmlhdGlvbiBpbmRpY2F0ZXMgdGhlIHR5cGljYWwgZGV2aWF0aW9uIGZyb20gdGhlIGxvY2FsIG1lYW4uIEFzIGV4cGVjdGVkLCB0aGUgc3RhbmRhcmQgZGV2aWF0aW9uIGxvY2FsbHkgaXMgdXN1YWxseSBsb3dlciB0aGFuIHRoZSBzdGFuZGFyZCBkZXZpYXRpb24gb2YgdGhlIHNhbXBsZSwgYW5kIGl0IHRlbmRzIHRvIGJlIGxhcmdlciBmb3IgdGhlIHRhaWxzLCB0aGF0IGlzIHRoZSBsb2NhdGlvbnMgd2hlcmUgdGhlIHZhbHVlcyBhcmUgcmFyZS4NCg0KVGhlIGxvY2FsIHN0YW5kYXJkIGRldmlhdGlvbiBpcyBhIGNydWRlIGVzdGltYXRvciBvZiB0aGUgdW5jZXJ0YWludHksIGFnYWluIGJlY2F1c2Ugd2UgZG8gbm90IGtub3cgdGhlIHVuZGVybHlpbmcgZGlzdHJpYnV0aW9uLg0KDQpPdGhlciBhcHByb2FjaGVzIGJhc2VkIG9uIGJvb3RzdHJhcHBpbmcgY291bGQgYmUgaW1wbGVtZW50ZWQsIGJ1dCB0aGV5IGFyZSBiZXlvbmQgdGhlIHNjb3BlIG9mIHRoZSBwcmVzZW50IGRpc2N1c3Npb24uDQoNClRoZSBpc3N1ZXMgYXNzZXNzaW5nIHRoZSBsZXZlbCBvZiB1bmNlcnRhaW50eSBpbiB0aGUgcHJlZGljdGlvbnMgd2l0aCBWb3Jvbm9pIHBvbHlnb25zLCBJRFcsIGFuZCBrLXBvaW50IG1lYW5zIHJlZmxlY3QgdGhlIGZhY3QgdGhhdCB0aGVzZSBtZXRob2RzIHdlcmUgbm90IGRlc2lnbmVkIHRvIGRlYWwgZXhwbGljaXRseSB3aXRoIHRoZSByYW5kb20gbmF0dXJlIG9mIHByZWRpY3RpbmcgZmllbGRzLg0KDQpPdGhlciBtZXRob2RzIGRlYWwgd2l0aCB0aGlzIGlzc3VlIG1vcmUgbmF0dXJhbGx5LiBXZSB3aWxsIHJldmlzaXQgdHdvIGVzdGltYXRpb24gbWV0aG9kcyB0aGF0IHdlcmUgY292ZXJlZCBiZWZvcmUsIGFuZCBzZWUgaG93IHRoZXkgY2FuIGJlIGFwcGxpZWQgdG8gc3BhdGlhbCBpbnRlcnBvbGF0aW9uLg0KDQojIFRyZW5kIHN1cmZhY2UgYW5hbHlzaXMNCg0KVHJlbmQgc3VyZmFjZSBhbmFseXNpcyBpcyBhIGZvcm0gb2YgbXVsdGl2YXJpYXRlIHJlZ3Jlc3Npb24gdGhhdCB1c2VzIHRoZSBjb29yZGluYXRlcyBvZiB0aGUgb2JzZXJ2YXRpb25zIHRvIGZpdCBhIHN1cmZhY2UgdG8gdGhlIGRhdGEuDQoNCkxldHMgc2VlIGhvdyB0aGlzIHdvdWxkIHdvcmsgd2l0aCBhIHNpbXVsYXRlZCBleGFtcGxlLiBXZSB3aWxsIGJlZ2luIGJ5IHNpbXVsYXRpbmcgYSBzZXQgb2Ygb2JzZXJ2YXRpb25zLCBiZWdpbm5pbmcgd2l0aCB0aGUgY29vcmRpbmF0ZXMgaW4gdGhlIHNxdWFyZSB1bml0IHJlZ2lvbjoNCmBgYHtyfQ0KbiA8LSAxODANCmRmIDwtIGRhdGEuZnJhbWUoeCA9IHJ1bmlmKG4gPSBuLCBtaW4gPSAwLCBtYXggPSAxKSwgDQogICAgICAgICAgICAgICAgIHkgPSBydW5pZihuID0gbiwgbWluID0gMCwgbWF4ID0gMSkpDQpgYGANCg0KUGxvdCB0aGVzZSBsb2NhdGlvbnM6DQpgYGB7cn0NCmdncGxvdChkYXRhID0gZGYsIGFlcyh4ID0geCwgeSA9IHkpKSArIA0KICBnZW9tX3BvaW50KCkgKyANCiAgY29vcmRfZXF1YWwoKQ0KYGBgDQoNCkxldHMgbm93IHNpbXVsYXRlIGEgc3BhdGlhbCBwcm9jZXNzIGFzIGZvbGxvd3M6DQpgYGB7cn0NCmRmIDwtIG11dGF0ZShkZiwgeiA9IDAuNSArIDAuMyAqIHggKyAwLjcgKiB5ICsgcm5vcm0obiA9IG4sIG1lYW4gPSAwLCBzZCA9IDAuMSkpDQpgYGANCg0KQSAzRCBzY2F0dGVycGxvdCBjYW4gYmUgdXNlZnVsIHRvIGV4cGxvcmUgdGhlIGRhdGE6DQpgYGB7cn0NCnBsb3RfbHkoZGF0YSA9IGRmLCB4ID0gfngsIHkgPSB+eSwgeiA9IH56LCBjb2xvciA9IH56KSAlPiUgDQogIGFkZF9tYXJrZXJzKCkgJT4lIA0KICBsYXlvdXQoc2NlbmUgPSBsaXN0KA0KICAgIGFzcGVjdG1vZGUgPSAibWFudWFsIiwgYXNwZWN0cmF0aW8gPSBsaXN0KHg9MSwgeT0xLCB6PTEpKSkNCmBgYA0KDQpXZSBjYW4gZml0IGEgdHJlbmQgc3VyZmFjZSB0byB0aGUgZGF0YSBhcyBmb2xsb3dzLiBJbiB0aGlzIGNhc2UsIHRoZSB0cmVuZCBpcyBsaW5lYXI6DQpgYGB7cn0NCnRyZW5kLmwgPC0gbG0oZm9ybXVsYSA9IHogfiB4ICsgeSwgZGF0YSA9IGRmKQ0Kc3VtbWFyeSh0cmVuZC5sKQ0KYGBgDQoNCkdpdmVuIGEgdHJlbmQgc3VyZmFjZSBtb2RlbCwgd2UgY2FuIGVzdGltYXRlIHRoZSB2YWx1ZSBvZiB0aGUgdmFyaWFibGUgJHokIGF0IGxvY2F0aW9ucyB3aGVyZSBpdCB3YXMgbm90IG1lYXN1cmVkLiBUeXBpY2FsbHkgdGhpcyBpcyBkb25lIGJ5IGludGVycG9sYXRpbmcgb24gYSBmaW5lIGdyaWQgdGhhdCBjYW4gYmUgdXNlZCBmb3Igdml6dWFsaXphdGlvbiBvciBmdXJ0aGVyIGFuYWx5c2lzLCBhcyBzaG93biBuZXh0Lg0KDQpXZSB3aWxsIGJlZ2luIGJ5IGNyZWF0aW5nIGEgZ3JpZCBmb3IgaW50ZXJwb2xhdGlvbi4gV2Ugd2lsbCBjYWxsIHRoZSBjb29yZGluYXRlcyBgeC5wYCBhbmQgYHkucGAuIFRoZXNlIHdlIGdlbmVyYXRlIGJ5IGNyZWF0aW5nIGEgc2VxdWVuY2Ugb2YgdmFsdWVzIGluIHRoZSBkb21haW4gb2YgdGhlIGRhdGEsIGZvciBpbnN0YW5jZSBpbiB0aGUgWzAsMV0gaW50ZXJ2YWw6DQpgYGB7cn0NCngucCA8LSBzZXEoZnJvbSA9IDAuMCwgdG8gPSAxLjAsIGJ5ID0gMC4wNSkNCnkucCA8LSBzZXEoZnJvbSA9IDAuMCwgdG8gPSAxLjAsIGJ5ID0gMC4wNSkNCmBgYA0KDQpGb3IgcHJlZGljdGlvbiwgd2Ugd2FudCBhbGwgY29tYmluYXRpb25zIG9mIGB4LnBgIGFuZCBgeS5wYCwgc28gd2UgZXhwYW5kIHRoZXNlIHR3byB2ZWN0b3JzIGludG8gYSBncmlkLCBieSBtZWFucyBvZiB0aGUgZnVuY3Rpb24gYGV4cGFuZC5ncmlkYDoNCmBgYHtyfQ0KZGYucCA8LSBleHBhbmQuZ3JpZCh4ID0geC5wLCB5ID0geS5wKQ0KYGBgDQoNCk5vdGljZSB0aGF0IHdoaWxlIGB4LnBgIGFuZCBgeS5wYCBhcmUgb2YgdmVjdG9ycyBvZiBzaXplIDIxLCB0aGUgZGF0YWZyYW1lIGBkZi5wYCBjb250YWlucyBge3J9MjEgKiAyMWAgb2JzZXJ2YXRpb25zLg0KDQpPbmNlIHRoYXQgd2UgaGF2ZSB0aGUgY29vcmRpbmF0ZXMgZm9yIGludGVycG9sYXRpb24sIHRoZSBgcHJlZGljdGAgZnVuY3Rpb24gY2FuIGJlIHVzZWQgaW4gY29uanVuY3Rpb24gd2l0aCB0aGUgcmVzdWx0cyBvZiB0aGUgZXN0aW1hdGlvbi4gV2hlbiBpbnZva2luZyB0aGUgZnVuY3Rpb24sIHdlIGluZGljYXRlIHRoYXQgd2Ugd2lzaCB0byBvYnRhaW4gYXMgd2VsbCB0aGUgc3RhbmRhcmQgZXJyb3JzIG9mIHRoZSBmaXR0ZWQgdmFsdWVzIChgc2UuZml0ID0gVFJVRWApLCBhcyB3ZWxsIGFzIHRoZSBpbnRlcnZhbCBvZiB0aGUgcHJlZGljdGlvbnMgYXQgYSA5NSUgbGV2ZWwgb2YgY29uZmlkZW5jZToNCmBgYHtyfQ0KcHJlZHMgPC0gcHJlZGljdCh0cmVuZC5sLCBuZXdkYXRhID0gZGYucCwgc2UuZml0ID0gVFJVRSwgaW50ZXJ2YWwgPSAicHJlZGljdGlvbiIsIGxldmVsID0gMC45NSkNCmBgYA0KDQpUaGUgaW50ZXJ2YWwgb2YgY29uZmlkZW5jZSBfb2YgdGhlIHByZWRpY3Rpb25zXyBhdCB0aGUgOTUlIGxldmVsIG9mIGNvbmZpZGVuY2UsIGluIHRoZSBmb3JtIG9mIHRoZSBsb3dlciAoYGx3cmApIGFuZCB1cHBlciAoYHVwcmApIGJvdW5kczoNCmBgYHtyfQ0Kc3VtbWFyeShwcmVkcyRmaXQpDQpgYGANCg0KVGhlc2UgdmFsdWVzIGluZGljYXRlIHRoYXQgdGhlIHByZWRpY3Rpb25zIG9mICR6X3AkIGFyZSwgd2l0aCA5NSUgb2YgY29uZmlkZW5jZSwgaW4gdGhlIGZvbGxvd2luZyBpbnRlcnZhbDoNCiQkDQpDSV97el9wfSA9IFt6X3twX3tsd3J9fSwgel97cF97dXByfX1dLg0KJCQNCg0KQSBjb252ZW5pZW50IHdheSB0byB2aXN1YWxpemUgdGhlIHJlc3VsdHMgb2YgdGhlIGFuYWx5c2lzIGFib3ZlLCB0aGF0IGlzLCB0byBpbnNwZWN0IHRoZSB0cmVuZCBzdXJmYWNlIGFuZCB0aGUgaW50ZXJ2YWwgb2YgY29uZmlkZW5jZSBvZiB0aGUgcHJlZGljdGlvbnMsIGlzIGJ5IG1lYW5zIG9mIGEgM0QgcGxvdCBhcyBmb2xsb3dzLg0KDQpGaXJzdCBjcmVhdGUgbWF0cmljZXMgd2l0aCB0aGUgcG9pbnQgZXN0aW1hdGVzIG9mIHRoZSB0cmVuZCBzdXJmYWNlIChgei5wYCksIGFuZCB0aGUgbG93ZXIgYW5kIHVwcGVyIGJvdW5kcyAoYHoucF9sYCwgYHoucF91YCk6DQpgYGB7cn0NCnoucCA8LSBtYXRyaXgoZGF0YSA9IHByZWRzJGZpdFssMV0sIG5yb3cgPSAyMSwgbmNvbCA9IDIxLCBieXJvdyA9IFRSVUUpDQp6LnBfbCA8LSBtYXRyaXgoZGF0YSA9IHByZWRzJGZpdFssMl0sIG5yb3cgPSAyMSwgbmNvbCA9IDIxLCBieXJvdyA9IFRSVUUpDQp6LnBfdSA8LSBtYXRyaXgoZGF0YSA9IHByZWRzJGZpdFssM10sIG5yb3cgPSAyMSwgbmNvbCA9IDIxLCBieXJvdyA9IFRSVUUpDQpgYGANCg0KVGhlIHBsb3QgaXMgY3JlYXRlZCB1c2luZyB0aGUgY29vcmRpbmF0ZXMgdXNlZCBmb3IgaW50ZXJwb2xhdGlvbiAoYHgucGAgYW5kIGB5LnBgKSBhbmQgdGhlIG1hdHJpY2VzIHdpdGggdGhlIHBvaW50IGVzdGltYXRlcyBgei5wYCBhbmQgdGhlIHVwcGVyIGFuZCBsb3dlciBib3VuZHMuIFRoZSB0eXBlIG9mIHBsb3QgaW4gdGhlIHBhY2thZ2UgYHBsb3RseWAgaXMgYSBfc3VyZmFjZV86DQpgYGB7cn0NCnRyZW5kLnBsb3QgPC0gcGxvdF9seSh4ID0gfngucCwgeSA9IH55LnAsIHogPSB+ei5wLCANCiAgICAgICAgdHlwZSA9ICJzdXJmYWNlIiwgY29sb3JzID0gIllsT3JSZCIpICU+JSANCiAgYWRkX3N1cmZhY2UoeCA9IH54LnAsIHkgPSB+eS5wLCB6ID0gfnoucF9sLCANCiAgICAgICAgICAgICAgb3BhY2l0eSA9IDAuNSwgc2hvd3NjYWxlID0gRkFMU0UpICU+JQ0KICBhZGRfc3VyZmFjZSh4ID0gfngucCwgeSA9IH55LnAsIHogPSB+ei5wX3UsIA0KICAgICAgICAgICAgICBvcGFjaXR5ID0gMC41LCBzaG93c2NhbGUgPSBGQUxTRSkgJT4lIA0KICBsYXlvdXQoc2NlbmUgPSBsaXN0KA0KICAgIGFzcGVjdG1vZGUgPSAibWFudWFsIiwgYXNwZWN0cmF0aW8gPSBsaXN0KHggPSAxLCB5ID0gMSwgeiA9IDEpKSkNCnRyZW5kLnBsb3QgIyU+JSBhZGRfbWFya2VycyhkYXRhID0gZGYsIHggPSB+eCwgeSA9IH55LCB6ID0gfnosIGNvbG9yID0gfnJlc2lkdWFsKQ0KYGBgDQoNCkluIHRoaXMgd2F5LCB3ZSBoYXZlIG5vdCBvbmx5IGFuIGVzdGltYXRlIG9mIHRoZSB1bmRlcmx5aW5nIGZpZWxkLCBidXQgYWxzbyBhIG1lYXN1cmUgb2YgdW5jZXJ0YWludHkgZm9yIG91ciBwcmVkaWN0aW9ucywgc2luY2Ugb3VyIGVzdGltYXRlZCB2YWx1ZXMgYXJlIGJvdW5kLCB3aXRoIDk1JSBjb25maWRlbmNlLCBiZXR3ZWVuIHRoZSBsb3dlciBhbmQgdXBwZXIgc3VyZmFjZXMuDQoNCkl0IGlzIGltcG9ydGFudCB0byBub3RlIHRoYXQsIGFsdGhvdWdoIHRoZSBjb25maWRlbmNlIGludGVydmFsIHByb3ZpZGVzIGEgbWVhc3VyZSBvZiB1bmNlcnRhaW50eSwgaXQgZG9lcyBub3QgcHJvdmlkZSBhbiBlc3RpbWF0ZSBvZiB0aGUgcHJlZGljdGlvbiBlcnJvciAkXGhhdHtcZXBzaWxvbn1fcCQuIFRoaXMgcXVhbnRpdHkgY2Fubm90IGJlIGNhbGN1bGF0ZWQgZGlyZWN0bHksIGJlY2F1c2Ugd2UgX2RvIG5vdCBrbm93IHRoZSB0cnVlIHZhbHVlIG9mIHRoZSBmaWVsZCBhdCBsb2NhdGlvbiAkcCRfLiBXZSB3aWxsIHJldmlzaXQgdGhpcyBwb2ludCBsYXRlci4NCg0KRm9yIHRoZSB0aW1lIGJlaW5nLCBsZXRzIGFwcGx5IHRyZW5kIHN1cmZhY2UgYW5hbHlzaXMgdG8gdGhlIFdhbGtlciBMYWtlIGRhdGEuDQoNCldlIHdpbGwgZmlyc3QgY2FsY3VsYXRlIHRoZSBwb2x5bm9taWFsIHRlcm1zIG9mIHRoZSBjb29yZGluYXRlcywgZm9yIGluc3RhbmNlIHRvIHRoZSAzcmQgZGVncmVlICh0aGlzIGNhbiBiZSBkb25lIHRvIGFueSBhcmJpdHJhcnkgZGVncmVlLCBob3dldmVyIGtlZXBpbmcgaW4gbWluZCB0aGUgY2F2ZWF0cyBkaXNjdXNzZWQgcHJldmlvdXNseSB3aXRoIHJlc3BlY3QgdG8gdHJlbmQgc3VyZmFjZSBhbmFseXNpcyk6DQpgYGB7cn0NCldhbGtlcl9MYWtlLnQgPC0gbXV0YXRlKFdhbGtlcl9MYWtlLnQsDQogICAgICAgICAgICAgICAgICAgICAgICBYMyA9IFheMywgWDJZID0gWF4yICogWSwgWDIgPSBYXjIsIA0KICAgICAgICAgICAgICAgICAgICAgICAgWFkgPSBYICogWSwNCiAgICAgICAgICAgICAgICAgICAgICAgIFkyID0gWV4yLCBYWTIgPSBYICogWV4yLCBZMyA9IFleMykNCmBgYA0KDQpXZSBjYW4gcHJvY2VlZCB0byBlc3RpbWF0ZSB0aGUgZm9sbG93aW5nIG1vZGVscy4NCg0KTGluZWFyIHRyZW5kIHN1cmZhY2UgbW9kZWw6DQpgYGB7cn0NCldMLnRyZW5kMSA8LSBsbShmb3JtdWxhID0gViB+IFggKyBZLCBkYXRhID0gV2Fsa2VyX0xha2UudCkNCnN1bW1hcnkoV0wudHJlbmQxKQ0KYGBgDQoNClF1YWRyYXRpYyB0cmVuZCBzdXJmYWNlIG1vZGVsOg0KYGBge3J9DQpXTC50cmVuZDIgPC0gbG0oZm9ybXVsYSA9IFYgfiBYMiArIFggKyBYWSArIFkgKyBZMiwgZGF0YSA9IFdhbGtlcl9MYWtlLnQpDQpzdW1tYXJ5KFdMLnRyZW5kMikNCmBgYA0KDQpDdWJpYyB0cmVuZCBzdXJmYWNlIG1vZGVsOg0KYGBge3J9DQpXTC50cmVuZDMgPC0gbG0oZm9ybXVsYSA9IFYgfiBYMyArIFgyWSArIFgyICsgWCArIFhZICsgWSArIFkyICsgWFkyICsgWTMsIA0KICAgICAgICAgICAgICAgIGRhdGEgPSBXYWxrZXJfTGFrZS50KQ0Kc3VtbWFyeShXTC50cmVuZDMpDQpgYGANCg0KSW5zcGVjdGlvbiBvZiB0aGUgcmVzdWx0cyBvZiB0aGUgdGhyZWUgbW9kZWxzIGFib3ZlIHN1Z2dlc3RzIHRoYXQgdGhlIGN1YmljIHRyZW5kIHN1cmZhY2UgcHJvdmlkZXMgdGhlIGJlc3QgZml0LCB3aXRoIHRoZSBoaWdoZXN0IGFkanVzdGVkIGNvZWZmaWNpZW50IG9mIGRldGVybWluYXRpb24sIGV2ZW4gaWYgdGhlIHZhbHVlIGlzIHJlbGF0aXZlbHkgbG93IGF0IGFwcHJveGltYXRlbHkgMC4xNi4gQWxzbywgdGhlIGN1YmljIHRyZW5kIHlpZWxkcyB0aGUgc21hbGxlc3Qgc3RhbmRhcmQgZXJyb3IsIHdoaWNoIGltcGxpZXMgdGhhdCB0aGUgaW50ZXJ2YWxzIG9mIGNvbmZpZGVuY2UgYXJlIHRpZ2h0ZXIsIGFuZCBoZW5jZSB0aGUgZGVncmVlIG9mIHVuY2VydGFpbnR5IGlzIHNtYWxsZXIuDQoNCkxldHMgY29tcGFyZSB0d28gb2YgdGhlc2UgbW9kZWxzIHRvIHNlZSBob3cgd2VsbCB0aGV5IGZpdCB0aGUgZGF0YS4NCg0KRmlyc3QsIHdlIHVzZSBjcmVhdGUgYW4gaW50ZXJwb2xhdGlvbiBncmlkLiBTdW1tYXJpemUgdGhlIGluZm9ybWF0aW9uIHRvIGFzY2VydGFpbiB0aGUgZG9tYWluIG9mIHRoZSBkYXRhOg0KYGBge3J9DQpzdW1tYXJ5KFdhbGtlcl9MYWtlLnRbLDI6M10pDQpgYGANCg0KV2UgY2FuIHNlZSB0aGF0IHRoZSBzcGF0aWFsIGRvbWFpbiBpcyBpbiB0aGUgcmFuZ2Ugb2YgWzgsMjUxXSBpbiBYLCBhbmQgWzgsMjkxXSBpbiBZLiBCYXNlZCBvbiB0aGlzIGluZm9ybWF0aW9uLCB3ZSBnZW5lcmF0ZSB0aGUgZm9sbG93aW5nIHNlcXVlbmNlIHRoYXQgdGhlbiBpcyBleHBhbmRlZCBpbnRvIGEgZ3JpZCBmb3IgcHJlZGljdGlvbjoNCmBgYHtyfQ0KWC5wIDwtIHNlcShmcm9tID0gMC4wLCB0byA9IDI1NS4wLCBieSA9IDIuNSkNClkucCA8LSBzZXEoZnJvbSA9IDAuMCwgdG8gPSAyOTUuMCwgYnkgPSAyLjUpDQpkZi5wIDwtIGV4cGFuZC5ncmlkKFggPSBYLnAsIFkgPSBZLnApDQpgYGANCg0KVG8gdGhpcyBkYXRhZnJhbWUgd2UgYWRkIHRoZSBwb2x5bm9taWFsIHRlcm1zOg0KYGBge3J9DQpkZi5wIDwtIG11dGF0ZShkZi5wLCBYMyA9IFheMywgWDJZID0gWF4yICogWSwgWDIgPSBYXjIsIA0KICAgICAgICAgICAgICAgWFkgPSBYICogWSwgDQogICAgICAgICAgICAgICBZMiA9IFleMiwgWFkyID0gWCAqIFleMiwgWTMgPSBZXjMpDQpgYGANCg0KVGhlIGludGVycG9sYXRlZCBxdWFkcmF0aWMgc3VyZmFjZSBpcyB0aGVuIG9idGFpbmVkIGFzOg0KYGBge3J9DQpXTC5wcmVkczIgPC0gcHJlZGljdChXTC50cmVuZDIsIG5ld2RhdGEgPSBkZi5wLCBzZS5maXQgPSBUUlVFLCBpbnRlcnZhbCA9ICJwcmVkaWN0aW9uIiwgbGV2ZWwgPSAwLjk1KQ0KYGBgDQoNCldoZXJlYXMgdGhlIGludGVycG9sYXRlZCBjdWJpYyBzdXJmYWNlIGlzIG9idGFpbmVkIGFzOg0KYGBge3J9DQpXTC5wcmVkczMgPC0gcHJlZGljdChXTC50cmVuZDMsIG5ld2RhdGEgPSBkZi5wLCBzZS5maXQgPSBUUlVFLCBpbnRlcnZhbCA9ICJwcmVkaWN0aW9uIiwgbGV2ZWwgPSAwLjk1KQ0KYGBgDQoNClRoZSBwcmVkaWN0aW9ucyBhcmUgdHJhbnNmb3JtZWQgaW50byBtYXRyaWNlcyBmb3IgcGxvdGluZy4NCg0KUXVhZHJhdGljIHRyZW5kIHN1cmZhY2UgYW5kIGxvd2VyIGFuZCB1cHBlciBib3VuZHMgb2YgdGhlIHByZWRpY3Rpb25zOg0KYGBge3J9DQp6LnAyIDwtIG1hdHJpeChkYXRhID0gV0wucHJlZHMyJGZpdFssMV0sIG5yb3cgPSAxMTksIG5jb2wgPSAxMDMsIGJ5cm93ID0gVFJVRSkNCnoucDJfbCA8LSBtYXRyaXgoZGF0YSA9IFdMLnByZWRzMiRmaXRbLDJdLCBucm93ID0gMTE5LCBuY29sID0gMTAzLCBieXJvdyA9IFRSVUUpDQp6LnAyX3UgPC0gbWF0cml4KGRhdGEgPSBXTC5wcmVkczIkZml0WywzXSwgbnJvdyA9IDExOSwgbmNvbCA9IDEwMywgYnlyb3cgPSBUUlVFKQ0KYGBgDQoNCkN1YmljIHRyZW5kIHN1cmZhY2UgYW5kIGxvd2VyIGFuZCB1cHBlciBib3VuZHMgb2YgdGhlIHByZWRpY3Rpb25zOg0KYGBge3J9DQp6LnAzIDwtIG1hdHJpeChkYXRhID0gV0wucHJlZHMzJGZpdFssMV0sIG5yb3cgPSAxMTksIG5jb2wgPSAxMDMsIGJ5cm93ID0gVFJVRSkNCnoucDNfbCA8LSBtYXRyaXgoZGF0YSA9IFdMLnByZWRzMyRmaXRbLDJdLCBucm93ID0gMTE5LCBuY29sID0gMTAzLCBieXJvdyA9IFRSVUUpDQp6LnAzX3UgPC0gbWF0cml4KGRhdGEgPSBXTC5wcmVkczMkZml0WywzXSwgbnJvdyA9IDExOSwgbmNvbCA9IDEwMywgYnlyb3cgPSBUUlVFKQ0KYGBgDQoNClRoaXMgaXMgdGhlIHF1YWRyYXRpYyB0cmVuZCBzdXJmYWNlIHdpdGggaXRzIGNvbmZpZGVuY2UgaW50ZXJ2YWwgb2YgcHJlZGljdGlvbnM6DQpgYGB7cn0NCldMLnBsb3QyIDwtIHBsb3RfbHkoeCA9IH5YLnAsIHkgPSB+WS5wLCB6ID0gfnoucDIsIA0KICAgICAgICB0eXBlID0gInN1cmZhY2UiLCBjb2xvcnMgPSAiWWxPclJkIikgJT4lIA0KICBhZGRfc3VyZmFjZSh4ID0gflgucCwgeSA9IH5ZLnAsIHogPSB+ei5wMl9sLCANCiAgICAgICAgICAgICAgb3BhY2l0eSA9IDAuNSwgc2hvd3NjYWxlID0gRkFMU0UpICU+JQ0KICBhZGRfc3VyZmFjZSh4ID0gflgucCwgeSA9IH5ZLnAsIHogPSB+ei5wMl91LCANCiAgICAgICAgICAgICAgb3BhY2l0eSA9IDAuNSwgc2hvd3NjYWxlID0gRkFMU0UpICU+JSANCiAgbGF5b3V0KHNjZW5lID0gbGlzdCgNCiAgICBhc3BlY3Rtb2RlID0gIm1hbnVhbCIsIGFzcGVjdHJhdGlvID0gbGlzdCh4ID0gMSwgeSA9IDEsIHogPSAxKSkpDQpXTC5wbG90Mg0KYGBgDQoNCkFuZCwgdGhpcyBpcyB0aGUgY3ViaWMgdHJlbmQgc3VyZmFjZSB3aXRoIGl0cyBjb25maWRlbmNlIGludGVydmFsIG9mIHByZWRpY3Rpb25zOg0KYGBge3J9DQpXTC5wbG90MyA8LSBwbG90X2x5KHggPSB+WC5wLCB5ID0gflkucCwgeiA9IH56LnAzLCANCiAgICAgICAgdHlwZSA9ICJzdXJmYWNlIiwgY29sb3JzID0gIllsT3JSZCIpICU+JSANCiAgYWRkX3N1cmZhY2UoeCA9IH5YLnAsIHkgPSB+WS5wLCB6ID0gfnoucDNfbCwgDQogICAgICAgICAgICAgIG9wYWNpdHkgPSAwLjUsIHNob3dzY2FsZSA9IEZBTFNFKSAlPiUNCiAgYWRkX3N1cmZhY2UoeCA9IH5YLnAsIHkgPSB+WS5wLCB6ID0gfnoucDNfdSwgDQogICAgICAgICAgICAgIG9wYWNpdHkgPSAwLjUsIHNob3dzY2FsZSA9IEZBTFNFKSAlPiUgDQogIGxheW91dChzY2VuZSA9IGxpc3QoDQogICAgYXNwZWN0bW9kZSA9ICJtYW51YWwiLCBhc3BlY3RyYXRpbyA9IGxpc3QoeCA9IDEsIHkgPSAxLCB6ID0gMSkpKQ0KV0wucGxvdDMNCmBgYA0KDQpBbGFzLCB0aGVzZSBtb2RlbHMgYXJlIG5vdCB2ZXJ5IHJlbGlhYmxlIGVzdGltYXRlcyBvZiB0aGUgdW5kZXJseWluZyBmaWVsZC4gQXMgY2FuIGJlIHNlZW4gZnJvbSB0aGUgcGxvdHMsIHRoZSBjb25maWRlbmNlIGludGVydmFscyBhcmUgZXh0cmVtZWx5IHdpZGUsIGFuZCBpbiBib3RoIGNhc2VzIGluY2x1ZGUgbmVnYXRpdmUgbnVtYmVycyBpbiB0aGUgbG93ZXIgYm91bmQuIFRoZSB1bmNlcnRhaW50eSBhc3NvY2lhdGVkIHdpdGggdGhlc2UgcHJlZGljdGlvbnMgaXMgcXVpdGUgc3Vic3RhbnRpYWwuDQoNCkFub3RoZXIgcXVlc3Rpb24sIGhvd2V2ZXIsIGlzIHdoZXRoZXIgdGhlIHBvaW50IGVzdGltYXRlcyBhcmUgYWNjdXJhdGUuIExldHMgYWRkIHRoZSBvYnNlcnZhdGlvbnMgdG8gdGhlIHBsb3Q6DQpgYGB7cn0NCldMLnBsb3QzICU+JQ0KICBhZGRfbWFya2VycyhkYXRhID0gV2Fsa2VyX0xha2UudCwgeCA9IH5YLCB5ID0gflksIHogPSB+ViwgDQogICAgICAgICAgICAgIGNvbG9yID0gflYsIG9wYWNpdHkgPSAwLjcsIHNob3dsZWdlbmQgPSBGQUxTRSkNCmBgYA0KDQpBbGFzLCB0aGUgdHJlbmQgc3VyZmFjZSBkb2VzIGEgbWVkaW9jcmUgam9iIHdpdGggdGhlIHBvaW50IGVzdGltYXRlcyBhcyB3ZWxsLg0KDQpBIHBvc3NpYmxlIHJlYXNvbiBmb3IgdGhpcyBpcyB0aGF0IHRoZSBtb2RlbCBmYWlsZWQgdG8gY2FwdHVyZSB0aGUgc3BhdGlhbCB2YXJpYWJpbGl0eS4gVG8gZXhwbG9yZSB0aGlzLCB3ZSB3aWxsIHBsb3QgdGhlIHJlc2lkdWFscyBvZiB0aGUgbW9kZWwsIGFmdGVyIGxhYmVsaW5nIHRoZW0gYXMgInBvc2l0aXZlIiBvciAibmVnYXRpdmUiOg0KYGBge3J9DQpXYWxrZXJfTGFrZS50JHJlc2lkdWFsMyA8LSBpZmVsc2UoV0wudHJlbmQzJHJlc2lkdWFscyA+IDAsICJQb3NpdGl2ZSIsICJOZWdhdGl2ZSIpDQpgYGANCg0KYGBge3J9DQpnZ3Bsb3QoZGF0YSA9IFdhbGtlcl9MYWtlLnQsIA0KICAgICAgIGFlcyh4ID0gWCwgeSA9IFksIGNvbG9yID0gcmVzaWR1YWwzKSkgKw0KICBnZW9tX3BvaW50KCkgKw0KICBjb29yZF9lcXVhbCgpDQpgYGANCg0KVmlzdWFsIGluc3BlY3Rpb24gb2YgdGhlIGRpc3RyaWJ1dGlvbiBvZiB0aGUgcmVzaWR1YWxzIHN0cm9uZ2x5IHN1Z2dlc3RzIHRoYXQgdGhleSBhcmUgbm90IHJhbmRvbS4gV2UgY2FuIGNoZWNrIHRoaXMgYnkgbWVhbnMgb2YgTW9yYW4ncyBJIGNvZWZmaWNpZW50LCBpZiB3ZSBjcmVhdGUgYSBsaXN0IG9mIHNwYXRpYWwgd2VpZ2h0cyBhcyBmb2xsb3dzOg0KYGBge3J9DQpXTC5saXN0dyA8LSBuYjJsaXN0dyhrbm4ybmIoa25lYXJuZWlnaChhcy5tYXRyaXgoV2Fsa2VyX0xha2UudFssMjozXSksIGsgPSA1KSkpDQpgYGANCg0KVGhlIHJlc3VsdHMgb2YgdGhlIGF1dG9jb3JyZWxhdGlvbiBhbmFseXNpcyBvZiB0aGUgcmVzaWR1YWxzIGFyZToNCmBgYHtyfQ0KbW9yYW4udGVzdCh4ID0gV0wudHJlbmQzJHJlc2lkdWFscywgbGlzdHcgPSBXTC5saXN0dykNCmBgYA0KDQpHaXZlbiB0aGUgbG93IHAtdmFsdWUsIHdlIGZhaWwgdG8gcmVqZWN0IHRoZSBudWxsIGh5cG90aGVzaXMsIGFuZCBjb25jbHVkZSwgd2l0aCBhIGhpZ2ggbGV2ZWwgb2YgY29uZmlkZW5jZSwgdGhhdCB0aGUgcmVzaWR1YWxzIGFyZSBub3QgaW5kZXBlbmRlbnQuIFRoaXMgaGFzIGltcG9ydGFudCBpbXBsaWNhdGlvbnMgZm9yIHNwYXRpYWwgaW50ZXJwb2xhdGlvbiwgYXMgd2Ugd2lsbCBkaXNjdXNzIGluIHRoZSBmb2xsb3dpbmcgcHJhY3RpY2UuDQoNCiMgQWNjdXJhY3kgYW5kIHByZWNpc2lvbg0KDQpCZWZvcmUgY29uY2x1ZGluZyB0aGlzIHByYWN0aWNlLCBpdCBpcyB3b3J0aHdoaWxlIHRvIG1ha2UgdGhlIGZvbGxvd2luZyBkaXN0aW5jdGlvbiBiZXR3ZWVuIGFjY3VyYWN5IGFuZCBwcmVjaXNpb24gb2YgdGhlIGVzdGltYXRlcy4NCg0KQWNjdXJhY3kgcmVmZXJzIHRvIGhvdyBjbG9zZSB0aGUgcHJlZGljdGVkIHZhbHVlcyAkXGhhdHt6fV9wJCBhcmUgdG8gdGhlIHRydWUgdmFsdWVzIG9mIHRoZSBmaWVsZC4gUHJlY2lzaW9uIHJlZmVycyB0byBob3cgbXVjaCB1bmNlcnRhaW50eSBpcyBhc3NvY2lhdGVkIHdpdGggc3VjaCBwcmVkaWN0aW9ucy4gTmFycm93IGludGVydmFscyBvZiBjb25maWRlbmNlIGltcGx5IGdyZWF0ZXIgcHJlY2lzaW9uLCB3aGVyZWFzIHRoZSBvcHBvc2l0ZSBpcyB0cnVlIHdoZW4gdGhlIGludGVydmFscyBvZiBjb25maWRlbmNlIGFyZSB3aWRlLg0KDQpBbiBleGFtcGxlIG9mIHRoZXNlIHR3byBwcm9wZXJ0aWVzIGlzIGFzIHNob3duIGluIEZpZ3VyZSAxLg0KDQohW0ZpZ3VyZSAxLiBBY2N1cmFjeSBhbmQgcHJlY2lzaW9uXSgxNi4gUHJhY3RpY2UgLSBGaWd1cmUgMS5qcGcpDQoNClBhbmVsIGEpIGluIHRoZSBmaWd1cmUgcmVwcmVzZW50cyBhIHNldCBvZiBhY2N1cmF0ZSBwb2ludHMsIHNpbmNlIHRoZXkgYXJlIG9uIGF2ZXJhZ2UgY2xvc2UgdG8gdGhlIG1hcmsuIEhvd2V2ZXIsIHRoZXkgYXJlIGltcHJlY2lzZSwgZ2l2ZW4gdGhlaXIgdmFyaWFiaWxpdHkuIFRoaXMgaXMgYWtpbiB0byBhIGdvb2QgcG9pbnQgZXN0aW1hdGUgdGhhdCBoYXMgd2lkZSBjb25maWRlbmNlIGludGVydmFscy4NCg0KUGFuZWwgYikgaXMgYSBzZXQgb2YgaW5hY2N1cmF0ZSBhbmQgaW1wcmVjaXNlIHBvaW50cy4NCg0KUGFuZWwgYykgaXMgYSBzZXQgb2YgcHJlY2lzZSBidXQgaW5hY2N1cmF0ZSBwb2ludHMuDQoNCkZpbmFsbHksIFBhbmVsIGQpIGlzIGEgc2V0IG9mIGFjY3VyYXRlIGFuZCBwcmVjaXNlIHBvaW50cy4NCg0KQWNjdXJhY3kgYW5kIHByZWNpc2lvbiBhcmUgaW1wb3J0YW50IGNyaXRlcmlhIHdoZW4gYXNzZXNzaW5nIHRoZSBxdWFsaXR5IG9mIGEgcHJlZGljdGl2ZSBtb2RlbC4NCg0KVGhpcyBjb25jbHVkZXMgUHJhY3RpY2UgMTUu